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ABSTRACT 



A virtual keyboard displayed on a computer display is used 
to emulate a conventional keyboard. The coordinate position 
of the keys on the virtual keyboard are stored as an array in 
the computer's memory. The key coordinates are ordered in 
the array based on their frequency of occurrence. A pointing 
device is used to select keys on the virtual keyboard. The 
input points generated by the pointing device are compared 
to the key coordinates stored in the key coordinate array one 
at a time. A predetermined bounding criteria is applied to 
determine whether the input point matches that key. If so, 
then the key is assigned to the input point and the search is 
discontinued. The search for a matching key continues until 
a match is found or until the end of the array is reached. 

13 Claims, 4 Drawing Sheets 
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IMAGE BASED KEYBOARD FOR A SMALL 
COMPUTING DEVICE 

FIELD OF THE INVENTION 

The present invention relates generally to input devices 
for a computing device, and more particularly, to a method 
for inputting data to a computing device using a virtual 
keyboard displayed on an electronic display. 

BACKGROUND OF THE INVENTION 

In a typical computer system, a keyboard and/or pointing 
device (such as a mouse) is used to enter data and commands 
into the computer. A monitor is used to display information 
to the user. In a desktop system, the keyboard and display are 
discrete components which are connected to the computer 
by means of a cable. In a laptop computer, the keyboard and 
display are designed as integral components of the computer 
case. In recent years, there has been interest in incorporating 
computers into small electronic devices such as cellular 
phones. 

While electronic display technology has advanced to the 
point that small display screens are readily commercially 
available, there are practical limits when it comes to reduc- 
ing the size of a keyboard. In small electronic devices, there 
may be insufficient space to incorporate a complete 101-key 
keypad as is found in a conventional desktop computer. 

One solution to this problem is to display an image of a 
conventional 101-key keyboard on a computer display. The 
user can select or "press" keys by means of a pointing device 
such as a mouse, track ball, light pen, or touch screen. The 
computer uses a coordinate system to map the input points 
to specific keys on the keyboard. In general, the computer 
maps the input points to the nearest key character which may 
span a few pixels in height and width. Mapping input points 
to the nearest key coordinate means that the distance to all 
keys must be found before a decision can be made about 
which key is closest to the input point. Computing the 
distance between input points and each of the keys each time 
a key is selected requires a large number of calculations to 
be performed. Thus, processor time is devoted to mapping 
the input points to individual keys which could be spent 
performing other tasks. 

SUMMARY OF THE INVENTION 

The present invention provides an improved method for 
mapping input points generated by a pointing device to 
individual keys of an on-screen keyboard. A keyboard is 
displayed to the user on an electronic display. The user 
selects one of the keys using a pointing device such as a 
mouse, or light pen. Key coordinates for each key are stored 
in a key coordinate array. The key coordinates are ordered 
based on some frequency criteria. That is, keys with higher 
probability of occurrence are placed in the key coordinate 
array before keys with a lower probability of occurrence. 
When a key is selected by the user, the computer compares 
the coordinates of the input point with the key coordinates 
stored in the key coordinate array beginning with the first 
array element (which has the highest frequency of 
occurrence) and continuing until a match is found. A bound 
check is performed to determine whether the input point 
matches any one of the keys in the array. For each array 
element the computer compares the coordinates of the input 
point to the key coordinates stored in the array. If the 
comparison meets some pre-defined condition, then a match 
is declared and the corresponding key is mapped to the 



,384 
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input. Otherwise, the bound check is performed for the next 
key in the key coordinate array. This process is repeated until 
a matching key is found, or until the end of the key 
coordinate array is reached in which case no mapping is 
5 performed. 

The present invention has several advantages over the 
methods used in the past. First, the use of the confidence 
intervals or bounds allows the key mapping search to be 
discontinued once a match is found. This greatly reduces the 

10 time needed to perform the key mapping search. Further 
reduction in search time is obtained by ordering the keys in 
the array based on frequency of occurrence. By doing so, it 
takes less time to match the input points to those keys used 
most frequently, and more time to keys used less frequently. 

15 The net result is a savings in search time. 

The present invention may, of course, be carried out in 
other specific ways than those herein set forth without 
departing from the spirit and essential characteristics of the 
invention. The present embodiments are, therefore, to be 

20 considered in all respects as illustrative and not restrictive, 
and all changes coming within the meaning and equivalency 
range of the appended claims are intended to be embraced 
therein. 

25 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic block diagram illustrating the main 
components of a computer system which uses the virtual 
keyboard interface. 
30 FIG. 2 is an illustration of a graphical user interface which 
implements the virtual keyboard. 

FIG. 3 is a flow diagram illustrating the key mapping 
method of the present invention. 

FIG. 4 is a flow diagram illustrating the scaling step of the 
key mapping method. 

DERAILED DESCRIPTION OF THE 
INVENTION 

40 Referring now to the drawings, a schematic block dia- 
gram illustrating the main components of a computer system 
is shown. The computer system includes a central processing 
unit (CPU) which operates according to instructions stored 
in its memory 14. An input/output (10) circuits 16 interfaces 

45 the CPU 12 with one or more input devices 18 such as a 
keyboard and/or pointing device and a display 20. Other 
output devices may also be connected to the CPU 12 via the 
I/O circuits 16. The computer may also include a mass 
storage device 22 such as a hard disk drive, floppy disk 

50 drive, or optical disk drive. 

In the present invention, it is contemplated that the input 
device 18 will comprise a pointing device such as a mouse 
which is used to move a cursor 48 within a graphical user 
interface (GUI) 30 displayed to the user on the display 20 

55 (see FIG. 2). The mouse, generates a position change signal 
in response to movement which is passed to the CPU 12. The 
position change signals are processed by the CPU 12 to 
cause a corresponding update of the position of a cursor 48 
on the display 20 of the computer system 10. The CPU 12 

60 keeps track of the position of the cursor 48. The mouse also 
includes buttons which can be pressed by the user. When the 
button is pressed down, a signal is generated and passed to 
the CPU 12. Similarly, another signal is sent by the mouse 
to the CPU 12 when the button is released. Pressing and 

65 releasing the button once is referred to as "clicking". Press- 
ing and releasing the mouse button twice is referred to as 
"double clicking". When the mouse is "clicked", the CPU 12 
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determines the coordinate position of the cursor and assigns is to store the coordinate position corresponding to the 

some meaning to the input based upon the coordinate center point for each key 54 in a key coordinate array and 

position of the cursor 48. then calculate the distance between the mouse click point 

A mouse or pointing device is most frequently used in a and each key 54 in the key coordinate array. The key closest 

graphical user interface (GUI) to enter commands. For 5 to the mouse click point would then be the selected key 54. 

example, the mouse may be used to select menu items by For a 101-key keyboard, this method would require 101 

positioning the cursor over the menu item and "clicking" separate calculations before a decision could be made, 

with the mouse. Programs can be launched using the mouse The present invention provides an improved method for 

by positioning the cursor over an icon (which represents the mapping the keys 54 of the virtual keyboard 52 to a mouse 

program) and "double clicking" with the mouse. In the past, 1° click or user input. To implement this method, the key 

the mouse or pointing device has not been commonly used coordinates for the keys 54 in the virtual keyboard 52 are 

to enter alphanumeric characters. stored in a key coordinate array in memory 14. The keys are 

Although the embodiment described herein employs a ordered in the key coordinate array based on some frequency 

conventional mouse, it will be appreciated that other types criteria. That is, coordinates of keys 54 with higher prob- 

of pointing devices may also be used such as a trackball, 15 ability of occurrence are present before keys 54 with lower 

digitizer tablet, light pen, or touch screen. All of these probability of occurrence. When a mouse click occurs, the 

devices are used to point to images displayed on the display mouse click coordinates are compared to the key coordinates 

20 and generate an input which is converted by the computer stored in the key coordinate array beginning with the first 

to a coordinate. The computer assigns meaning to the input element of the array and continuing until a match is found, 

based upon the coordinate position of the input point. 20 For each element of the key coordinate array, a bound check 

Referring now to FIG. 2, a graphical user interface 30 of * performed to determine whether the mouse click coordi- 

a user application running on the computer system 10 is ° ate matches lhat ke y £ thc , bou * d check cr " ena L 1S 

shown. The user application illustrated in FIG. 2 is a word satisfied then the corresponding key 54 is mapped to the 

processing application. It is to be understood, however, that nc mouse click and the search is stopped. If the bound check 

the present invention can be used with virtually any type of 25 criteria is not satisfied, then the bound check is performed 

user application and is not limited to word processing for the next key 54. This process contmues untU a matching 

applications. The GUI 30 differs from conventional GUI's in key 54 is found or until the end of the array is reached in 

that it provides for entry of alphanumeric characters or other whlch no mapping will occur, 

keyboard commands (such as function keys) by means of a As stated above, the CPU 12 performs a bound check to 

mouse or other pointing device. The GUI 30 is displayed to determine whether a given key 54 maps to a mouse click or 

the user on the screen of the computer display 20. other user input. In general, the CPU 12 determines whether 

The GUI 30 comprises a window 32 having a title bar 34 the x and y coordinate of the input point (i.e., mouse click 

across the top of the window 32. Title bar buttons 36 are ■ P<>int, pen point, touch point, etc.) fall within a certain range, 

disposed on the right hand of the title bar 34. These buttons called the confidence interval, of the key coordinate. For 

36 are used to close, maximize and minimize the window 32. example, assume that each letter key 54 in the virtual 

Buttons 36 are activated using the "point and click" method. keyboard 52 has a width and a height W y . The confi- 

These buttons will be familiar to users of the Windows 95 dence intervals may be chosen to be half the width and* 

and Windows NT operating systems. A status bar 56 where height of the letter key. That is, WJ2 and W/2 are the 

messages or icons are displayed for the user extends across , n confidence intervals. Let P^ Y k ) represent any key 

the bottom of the window coordinate in the key coordinate array and P(X, Y) represent 

Below the title bar 34 is the menu bar 38. The menu bar * e input point on the virtual keyboard 52 The bound check 

38 includes a tab strip 40 with a series of tabs 42. Menu for a vahd ke ? 15 calculated as follows: 

buttons 44 are placed on the tab strip 40. Different sets of |x-xJ^wj2 and 

menu buttons 44 can be selected by "clicking" on the tabs 42 45 

with the mouse. Menu buttons 44 are also selected by Iy-yJ^w/i 
"clicking" the menu button 44 with the mouse. 

The area below the menu bar 38 is the work area 46 where If the bound check criteria set forth above is satisfied, then 

the object being manipulated by a user application is dis- the key with the midpoint P*(X*, Y k ) is mapped to the input 

played to the user. The cursor 48, which is moved by the 50 point. On the other hand, the bound check criteria is not 

mouse, is shown in the work area 46. satisfied under the following conditions: 

Below the work area 46 is the keyboard panel 50 in which 

the image 52 of a keyboard is displayed. This virtual |X-XJ>W72oi 

keyboard 52 is used in combination with the mouse or other \y-y^>w/i 

pointing device to emulate the operation of a conventional 55 

keyboard. To use the virtual keyboard 52, the cursor 48 is In this case, the key is not mapped to the mouse click point, 

positioned over one of the keys 54 (i.e., pointed at the key) When the bound check criteria is not satisfied, the CPU 12 

of the virtual keyboard 52 and the mouse is "clicked". When moves to the next key coordinate in the key coordinate array 

a mouse click occurs within the boundary of the virtual and repeats the bound check. 

keyboard 52, the CPU 12 or other logic controller must 60 FIG. 3 is a flow chart illustrating the key mapping 

determine which key was selected. The CPU 12 then deter- algorithm of the present invention. The first step is to create 

mines what action to take based upon the key which was an array of key coordinates (block 100). This array is 

selected. The process of assigning a key to the mouse click referred to herein as the key coordinate array. The CPU 12 

is referred to as mapping. performs other tasks until user input, such as a mouse click, 
There are numerous ways in which to map the keys 54 of 65 is detected (block 102). Once the user input is detected, the 

the virtual' keyboard 52 to a mouse click point within the CPU 12 goes to the first element in the key coordinate array 

area of the virtual keyboard 52. Perhaps the simplest method (block 104). Using the key coordinates in the first element 
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of the key coordinate array, the CPU 12 performs a bound 
check (block 106). If the bound check is satisfied, the 
corresponding key is mapped to the mouse click (block 108) 
and stops the key mapping search (block 116). On the other 
hand, if the bound check criteria is not satisfied, the CPU 12 
determines whether it is on the last element of the key 
coordinate array (block 110). If not, the CPU 12 goes to the 
next element in the key coordinate array (block 112) and 
repeats the bound check (block 106). This process repeats 
until a matching key is found, or until the last element in the 
key coordinate array is reached. When the last element of the 
key coordinate array is reached without a match, then no 
mapping is performed (block 114) and the key mapping 
search is stopped (block 116). 

The confidence intervals used to perform the bound 
checks are based on the key shapes and sizes. As can be seen 
in FIG. 2, all of the keys 54 are of equal height but vary in 
width. The confidence intervals given above are based on the 
dimensions of the letter keys 54 which have the smallest 
width. The result is that those keys 54 wider than the letter 
keys 54 will have dead zones adjacent the outer ends thereof. 
This may prove firustrating to some users who "click" with 
the pointing device but do not get any corresponding 
response. In the preferred embodiment of the invention, this 
problem is solved by assigning widths to all keys 54 which 
are integer multiples of the letter keys. For example, the 
ENTER key is twice the width of the letter keys and the 
space bar is three times the width of the letter key. Keys 54 
which are twice the width of the letter keys will have two 
center points both of which are stored in the key coordinate 
array. Similarly, keys 54 which are three times as wide as the 
letter keys will have three center points which are, again, 
stored in the key coordinate array. Thus some keys 54 will 
be represented multiple times in the key coordinate array. In 
keys 54 with multiple center points, the distance between the 
center points is equal to the width of the letter keys. This 
arrangement eliminates the dead zones in the keys 54 which 
are wider than the letter keys. 

Another problem which may be encountered is when the 
user "clicks" on the boundary between two keys 54. Both 
keys 54 will satisfy the bound check criteria. In this case, the 
CPU 12 will assign the key 54 to the input which is appears 
first in the key coordinate array. This key 54 may not be the 
one which was intended by the user. It is reasonable to 
assume that this situation will occur infrequently since most 
key selections will occur near the center of the keys 54. 
When it does occur, the key 54 with the highest probability 
of occurrence will be selected thus minimizing the chance of 
an error. When an error is made, the user is likely to view the 
error as a typing mistake. 

In some cases, the virtual keyboard 52 may be contained 
in a scalable window 32 or other container. When the 
window 32 or container is resized, the virtual keyboard 52 
is scaled to fit into the resized • window 32. Thus, an 
additional scaling step is needed before a bound check can 
be performed. In the preferred embodiment, the key coor- 
dinates stored in the key coordinate array are for a standard 
keyboard image which corresponds to a prefixed container 
dimension. When a mouse click occurs, the mouse click 
point is scaled from the container dimension for the resized 
keyboard container to the prefixed keyboard container 
dimension before the bound check is performed. FIG. 4 
illustrates this scaling algorithm. At block 120, the CPU 12 
determines whether the keyboard is scalable. If so, the CPU 
12 obtains the keyboard container dimensions using a func- 
tion call to the operating system (block 122). After obtaining 
the keyboard container dimension, the mouse click point is 
scaled to the prefixed keyboard container dimension(block 
124). 
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From the forgoing, it is apparent that the present invention 
provides an improved method for performing a key mapping 
search. Ordering the keys based on frequency of occurrence 
increases the probability of finding an early match. Further, 
the confidence intervals or bounds enable this CPU 12 to 
break out of the key mapping search once a match is found. 
Thus, the need to calculate distance to all key points and then 
find the minimum distance is avoided. Moreover, the bound 
check criteria described is simpler than calculating the 
distance between two points which involves squaring. 

The present invention may, of course, be carried out in 
other specific ways than those herein set forth without 
departing from the spirit and essential characteristics of the 
invention. The present embodiments are, therefore, to be 
considered in all respects as illustrative and not restrictive, 
15 and all changes coining within the meaning and equivalency 
range of the appended claims are intended to be embraced 
therein. 
What is claimed is: 

1. In a computer system having a memory, a method for 
20 mapping keys of a virtual keyboard displayed on a computer 

display to an input point generated by a pointing device 
comprising: 

a) storing key coordinates corresponding to the keys on 
said virtual keyboard in a key coordinate array in said 

25 memory; 

b) comparing the coordinates of said input point to the key 
coordinates stored in said key coordinate array in a 
predetermined sequence until a matching key is found 
or the end of the key coordinate array is reached; 

c) for each given key coordinate, determining whether the 
coordinates of said input point satisfy a predetermined 
bounding criteria; and 

d) if said bounding criteria is satisfied, assigning the 
corresponding key to said input point. 

2. The key mapping method of claim 1 further including 
the step of ordering at least some of the key coordinates in 
said key coordinate array based upon frequency of occur- 
rence of the corresponding keys. 

3. The key mapping method according to claim 1 wherein 
40 said keyboard is displayed in a scalable keyboard container 

and further including the step of resizing said keyboard 
container. 

4. The key mapping method according to claim 3 further 
including step of scaling the input point to a prefixed 

45 keyboard container dimension before comparing the input 
point to said key coordinates. 

5. In a computer system having a memory, a method for 
mapping keys of a virtual keyboard displayed on a computer 
display to an input point generated by a pointing device 

50 comprising: 

a) storing key coordinates corresponding to keys on said 
virtual keyboard in" a key coordinate array in said 
memory; 

55 b) ordering at least some of the key coordinates in said 
key coordinate array based on the frequency of occur- 
rence of the corresponding keys; 

c) determining the coordinates of said input point; 

d) comparing the coordinates of said input point to the key 
coordinates stored in said key coordinate array in a 
predetermined sequence until a matching key is found 
or the end of the key coordinate array is reached; 

e) for each given key coordinate, determining whether the 
coordinates of said user input point satisfy a predeter- 
mined bounding criteria; and 

f) if said bounding criteria is satisfied, assigning the 
corresponding key to said user input. 
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6. The key mapping method according to claim 5 wherein 
said keyboard is displayed in a scalable keyboard container 
and further including the step of resizing said keyboard 
container. 

7. The key mapping method according to claim 6 further 5 
including step of scaling the input point to a prefixed 
keyboard container dimension before comparing the input 
point to said key coordinates. 

8. An input system for a computer comprising: 

a) a display for displaying a keyboard image comprising 10 
a plurality of keys; 

b) a pointing device for generating input points having 
coordinates referenced to the image displayed on said 
display; i5 

c) a memory for storing key coordinates corresponding to 
the keys in said keyboard image; and 

d) a logic device for: 

i) comparing the coordinates of said input point to the 
key coordinates stored in said memory in a prede- 2 o 
termined sequence until a matching key is found; 

ii) for each key coordinate, determining whether said 
input point satisfies a predetermined bounding cri- 
teria; 

iii) if said bounding criteria is satisfied, assigning the 
corresponding key to said input point. 



8 

9. The input system of claim 8 wherein said keyboard 
image is displayed on said display in a scalable keyboard 
container. 

10. The input system of claim 9 further including means 
for scaling said input points to a prefixed keyboard container 
dimension. 

11. An input system for a computer comprising: 

a) a display for displaying a keyboard image comprising 
a plurality of keys; 

b) a pointing device for generating input points within 
said keyboard image; 

c) a memory for storing key coordinates corresponding to 
the keys in said keyboard image; and 

d) bound check means for sequentially comparing the 
coordinates of said input point to said key coordinates 
stored in memory and applying a predetermined bound- 
ing criteria to determine whether the key corresponding 
to said key coordinate is the selected key, until a 
matching key is found. 

12. The input system of claim 11 wherein said keyboard 
image is displayed on said display in a scalable keyboard 
container. 

13. The input system of claim 12 further including means 
for scaling said input points to a prefixed keyboard container 
dimension. 

+ * * + * 
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